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Abstract. In this paper, a set of heuristics for evaluating the usability of 
mobile map applications is introduced. We developed the heuristics by ex- 
ploring the present generic heuristics and then forming new theory-based 
heuristics. Usability specialists tested the heuristics by evaluating the usa- 
bility of a mobile map application with both generic and domain-specific 
heuristics. As a result, more usability problems were found with the pro- 
posed domain-specific heuristics. In addition, based on the evaluators' 
views theinitial domain-specific heuristics were further developed. Wecon- 
clude by proposing domain-specific usability heuristics for evaluating the 
mobi I e map appl i cati ons. 
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1. Introduction 

Mobile map applications (MMAs) are a current trend of today. Nowadays 
not only IT oriented people, but also almost everyone who has a smart 
phone uses MMAs. However, currently usability problems concerning 
MMAs are not taken in proper consideration. One typical reason for these 
problems is that there are not enough resources, such as knowledge, ti me or 
money, to put i nto the usabi I i ty engi neeri ng of a map devel opment proj ect. 

Analyzing the application's usability with usability heuristics during the 
development phase is a low-cost and easily implementable way to improve 
usabi I i ty. The probl em though i s that the currentl y exi sti ng usabi lity heuris- 
tics are too general to be applicable for evaluating the usability of MMAs. 
Many research reports (e.g., Bertini et al. 2006) state that domain-specific 
usability heuristics have been more suitable for evaluating the usability of 
special applications than general heuristics such as Nielsen's set of usability 



heuristics (Nielsen &Molich 1990, Nielsen 1994). Domain-specific usability 
heuristics have been introduced for example to evaluate computer games 
(Pinelle et al. 2008), children's e-learning applications (Alsumait & Al- 
Osaimi 2009) and IT security management tools (J aferian et al. 2011). Still 
there are no suitable heuristics for MM As. To additionally evaluate, support 
and improve the usability of MMAs there is the need to develop domain- 
specifi c usabi I ity heuristics. 

I n this study, we test and propose a new set of usability heuristics to sup- 
port the development of MMAs. Firstly, the heuristics were derived via a 
theoretical-conceptual procedure based on theories from cartography 
(MacEachren 1995, Van Laar 2001 Nivala & Sarjakoski 2005, Tyner 2010, 
Krygier & Wood 2011) and visual design (Mullet & Sano 1995, Lidwell 
2003) in relation to the Nielsen's (1994) general heuristics. Special atten- 
tion was paid towards the visual design of the user interface (Ul) of map 
applications. This is due to the fact that mobile maps deploy specialized 
visual elements rendering it essential that the user understands their mean- 
ing. The proposed heuristics also cover information about the user's loca- 
tion, unambiguous route guidance, map scalability, adaptability of visible 
information depending on the device's screen size, up-to-date maps, appli- 
cation customizability to support user's personal interests and use of 
shortcuts to save i important I ocati ons. 

The rest of the paper is organized as follows. First, we discuss the back- 
ground of the topic, including the previously indicated usability problems 
and the usability inspection methods of MMAs. Additionally we highlight 
the gaps of current general usability heuristics. Second, we describe our 
research method. Third, we discuss the resulting initial domain-specific 
heuristics in detail. Finally, we conclude by describing our findings. 



2. Background 

The usability of MMAs is not trouble-free. The problems are related to, for 
example, the small screen size of the mobile device, interaction limitations 
of the device, connection speed and limited battery life. This is in addition 
to several visualization related issues, such as choice of level of detail, en- 
hancement effects, color choices, hierarchy in the use of symbols and visu- 
alization of off-screen information (Looijeetal. 2007). 

Although the problems have been pointed out, they typically still exist in the 
mobile map services. There are some reasons for this. For instance, map 
application developers are often not very familiar with usability related is- 
sues. Nivala et al. (2007) maintain that although usability is seen as an ad- 



vantage in the competition to be successful on the map application market, 
companies often fail to implement the usability engineering approach due 
to lack of knowledge and resources. Moreover, one problematic issue still is 
that usability has not played an important role in product development (J e- 
rome& Kazman 2005). 

Heuristics evaluation (HE) is a discount method for evaluating usability, as 
HE can be conducted with low resources in a short timeframe (Nielsen 
1992). Another advantage is that although the best results can be achieved 
with usability professionals performing HE, also non- professionals can use 
the method to evaluate product usability with some success (Baker et al. 
2002). HE has gained criticism for example concerning the variability and 
in-depth results depending on the evaluator's expertise and commitment 
(Hertzum &J acobsen 2003). De Kock et al. (2009) emphasize the type of 
information HE conveys compared to empirical user testing in that HE is 
focusing on identifying errors, whereas empirical user testing is determined 
by effectiveness, efficiency and user satisfaction. They also point out that as 
empirical user testing deals with the questions such as what and how, HE 
focuses on a meta-level which considers the questions why and when. 

The applicability of Nielsen's heuristics (1994) is limited in terms of evalu- 
ating MM As. This is because the heuristic set does not cover the specific 
aspects of MM As such as location awareness, mobility and interruptions. 
The visual design of mobile maps comprises multiple viewpoints to visual 
design (eg., Tyner 2010). Further, Nielsen's heuristics which relate to visu- 
al design do not properly take into account cartographic visual design prin- 
ci pi es. Therefore, these exi sti ng heuri sti cs I ack many essenti al vi ewpoi nts to 
effective visual mobile map design, and novel domain-specific heuristics are 
needed. 



3. Research Method 

The study was implemented in four phases. First, we explored the generic 
heuristics (Nielsen 1994) and pointed out their limitations for evaluating 
the usability of MMAs. Second, we familiarized ourselves with current do- 
main-specific heuristic sets (Pinelle et al. 2008, Alsumait & Al-Osaimi 
2009, J aferian et al. 201]) and explored which methods were used to devel- 
op these heuristics. The third phase was to formulate the new heuristics. 
We derived the initial usability heuristics from Nielsen's (1994) generic us- 
ability heuristics by using a conceptual -theoretical approach suitablefor the 
evaluation of MMAs. A similar approach has also been used in the devel- 
opment of other domain-specific usability heuristics (Alsumait & Al-Osaimi 
2009, 1 nostroza et al. 2012). 



Finally, the applicability of the initial usability heuristics for MMAs was 
tested. We asked four usability specialists to evaluate an M M A by complet- 
ing specific map-related tasks (self-locating, searching address, exploring 
the map, wayfinding and using POIs). Similar toj aferian et al. (2011) half 
of the evaluators were asked to use the original Nielsen heuristics to do the 
evaluation and half were asked to use our initial heuristics that are present- 
ed in Table 1 The evaluated application was NavFree (Navmii Holding pic. 
20 13). The application was chosen because it was not familiar to the evalua- 
tors, but is still widely used, offering common MM A functionalities. All four 
evaluators used a smart phone to do the evaluation: Apple iPhone 3GS (3), 
and an Android phone Samsung Galaxy Nexus (1). After completing the 
eval uati on, we asked the eval uators to rate the appl i cabi I ity of the heuri sti cs 
for evaluating the MM A and the intelligibility of each of the heuristics. We 
also asked for specific feedback of the heuristics. 



Jakob Nielsen's Heuristics 


Proposed Heuristics for Mobile Map Applica- 
tions 


1. Visibility of system status 

The system should always keep users in- 
formed about what is going on, through 
appropriate feedback within reasonable time. 


1. Visibility of the contextual map functions 

The map application should always keep the 
user informed about what is going on, through 
appropriate feedback within a reasonable time. 
The map functions should be visible. 


2. Match between system and the real 
world 

The system should speak the users' lan- 
guage, with words, phrases and concepts 
familiar to the user, rather than system- 
oriented terms. Follow real-world conven- 
tions, making information appear in a natural 
and logical order. 


2. Match between the system and the physi- 
cal surroundings of the user 

The map application should show clear indica- 
tion of the user's current location on the map 
and of the possible target location. It is essen- 
tial that the map compares in an understanda- 
ble way with the physical surroundings of the 
user. The map should be up-to-date. 


3. User control and freedom 

Users often choose system functions by 
mistake and will need a clearly marked 
"emergency exit" to leave the unwanted 
state without having to go through an ex- 
tended dialogue. Support undo and redo. 


3. User control over map functions and 
locations 

Allow the user to take control over map applica- 
tion when interruptions (from the mobile device: 
phone call, message, other applications' notifi- 
cations, from the concrete surroundings: traffic, 
weather, traffic lights) happen. 


4. Consistency and standards 

Users should not have to wonder whether 
different words, situations, or actions mean 
the same thing. Follow platform conventions. 


4. Consistency and standards 

Follow platform conventions. Use clear, intui- 
tive, commonly known map symbols. 


5. Error prevention 

Even better than good error messages is a 
careful design which prevents a problem 
from occurring in the first place. Either elimi- 
nate error-prone conditions or check for 
them and present users with a confirmation 
option before they commit to the action. 


5. Error prevention 

Even better than good error messages is a 
careful design, which prevents a problem from 
occurring in the first place. If errors still happen, 
make sure to offer the possibility to recover 
from them. 



6. Recognition rather than recall 

Minimize the user's memory load by making 
objects, actions, and options visible. The 
user should not have to remember infor- 
mation from one part of the dialogue to an- 
other. Instructions for use of the system 
should be visible or easily retrievable when- 
ever appropriate. 


6. Recognition rather than recall 

Make sure that the main functions of the map 
application (e.g. search, route guidance, zoom- 
ing, panning) are easily accessible. Use short 
menu paths for the main functions or keep the 
main functions visible at all times. 


7. Flexibility and efficiency of use 

Accelerators - unseen by the novice user - 
may often speed up the interaction for the 
expert user such that the system can cater 
to both inexperienced and experienced us- 
ers. Allow users to tailor frequent actions. 


7. Flexibility and efficiency of use 

Offer flexible options for the main map func- 
tions. Allow the user to save locations to be 
used as shortcuts (e.g. home) and support POI 
information. Give easy access to additional 
information (metadata, links, user-generated 
content). Make sure the user interface is scala- 
ble for different screen sizes of mobile devices. 


8. Aesthetic and minimalist design 

Dialogues should not contain information 
which is irrelevant or rarely needed. Every 
extra unit of information in a dialogue com- 
petes with the relevant units of information 
and diminishes their relative visibility. 


8. Balanced and simplistic visual design 

Harmonious overall appearance should consist 
of clear contrast between visual elements, 
balanced layout and informative colors. Avoid 
visual clutter. 


9. Help users recognize, diagnose, and 
recover from errors 

Error messages should be expressed in 
plain language (no codes), precisely indicate 
the problem, and constructively suggest a 
solution. 


9. Help users recognize, diagnose, and re- 
cover from errors 

Error messages should be expressed in plain 
language (no codes), precisely indicate the 
problem, and constructively suggest a solution. 
Indicate clearly the reasons if the searched 
locations are not found. Save the user's previ- 
ous searches for fast repetition. 


10. Help and documentation 

Even though it is better if the system can be 
used without documentation, it may be nec- 
essary to provide help and documentation. 
Any such information should be easy to 
search, focused on the user's task, list con- 
crete steps to be carried out, and not be too 
large. 


10. Help and documentation 

Even though it is better if the system can be 
used without documentation, it may be neces- 
sary to provide help and documentation. Pro- 
vide both: fast guidance focused on the user's 
task and a more detailed guide with search 
functions. 



Table L General and domain-specific heuristics 



4. Results 

The testing of the proposed MMA heuristics compared to the Nielsen (1994) 
heuristics pointed out that with the MMA heuristics more usability prob- 
lems (19, of which 6 were severe) were found than with the Nielsen heuris- 
tics (15, of which 5 were severe). On a scale from lto 5, where 1 is the least 
suitableand 5 is the most suitable, the suitability of theMMA heuristics was 
rated twice as 4 and the Nielsen heuristics as 2 and 3. 



The initial heuristics (Table 1} have been further elaborated by feedback of 
the applicability of the heuristics given by the usability specialists. In the 
following list we introduce the proposed usability heuristics for MMAs and 
present thejustifi cation for each of the heuristics. 

1 Visibility of the contextual map functions and important loca- 
tions. The map application should always interact with the user by giv- 
ing informative feedback within reasonable time. The map functions 
should be visible. The map view should constantly stay visible when the 
map application is in use. 

The limited display size of the mobile devices causes challenges for both 
setting map functions and making important map locations visible (Burigat 
et al. 2008). The power saving mode should not turn on if the user is in 
constant need of the map appl i cati on's assistance. 

2. Match between the system and the physical surroundings of 
the user. The map application should show clear indication of the us- 
er's location and other important locations (eg. destinations and POIs) 
on the map. It is essential that the map corresponds in an understanda- 
ble way with the physical surroundings of the user. The map should be 
up-to-date. 

Oulasvirta et al. (2008) have compared the embodied interaction of 2D ver- 
sus 3D mobile maps and summarized that 3D maps present realistic repre- 
sentation of objects, variable views from first- person perspective and more 
degrees of freedom in movement. 2D maps guide users into using environ- 
mental cues like street names and crossings. Mei linger et al. (2007) point 
out that different map types suit different tasks. 

3. User control over map functions. Allow the user to take control of 
the map appl i cati on when i nterrupti ons (from the mobi I e devi ce: phone 
call, message, other applications' notifications, from the concrete sur- 
roundings: traffic, weather, traffic lights) happen. Allow multitasking. 

Interruptions such as incoming emails, SMSs and phone calls influence 
interaction with mobile devices. When such interruptions occur, the appli- 
cation should save its current state and still beableto give the needed navi- 
gation instructions. As the use of a MMA is often concurrent with other 
tasks (Tamminen etal. 2004), allowing multitasking is essential. The MMA 
should also be context- aware, i.e. adapt to the surrounding conditions as 
that may enable more efficient uses of mobile applications (Hakkila et al. 
2009). For example, the MMA should adjust the visibility of screen size 
accordi ng to the I i ghti ng conditi ons and ti me of day. 



4. Consistency and standards. Follow platform conventions in the 
user interface design. Be consistent within the use of interaction ges- 
tures, controls, functions, elements of user interface and map features. 
Use clear, intuitive, commonly known map symbols. 

Symbols for mobile maps are designed for small screen size in which it is 
essentially important to consider clarity, intuitiveness and map symbol 
conventions. In order for the map symbols to be clear, simplification and 
abstraction are essential (Mullet & Sano 1995). Krygier and Wood (201]) 
emphasize that map symbols should work through resemblance, relation- 
ship, convention, difference, and standardization. 

5. E rror prevention. M ake the map appl i cati on free of errors. I f errors 
still happen, be sure to offer the possibility to recover from them. Pre- 
vent the user from getti ng I ost. 

Careful testing of the MM A should be performed in order to reduce the 
amount of errors. Wayfinding support prevents users from getting lost 
(Schmid et al. 2010) - as long as the wayfinding instructions are correct. 
Besides visual and audible instructions, also tactile feed back can be given to 
the users (Pielot et al. 2009). Tversky (2003) emphasizes the use of local 
environmental cues. 

6. Recognition rather than recall. Minimize the user's memory load. 
Make sure that the main functions of the map application (e.g. explor- 
ing, route guidance, zooming, panning, POI selection) are easily acces- 
sible. Use short menu paths for the main functions or keep the main 
functions present all the time. 

Mayer and Moreno (2003) discuss cognitive overload in the context of mul- 
timedia learning. To avoid overload, they suggest solutions, such as off- 
loading by using different multimedia channels to present information, 
providing cues about how the user can select and organize the data and 
aligning the content in a balanced way. 

7. Flexibility, scalability and efficiency of use. Offer flexible options 
for the main map functions. Allow the user to save locations to be used 
as shortcuts (e.g. home) and support POI information. Give easy access 
to additional information (metadata, links, user- generated content). 
M ake sure the user i nterface is scalable for different screen sizes of mo- 
bile devices. 

Setlur et al. (2010) present three types of optimizations implemented to 
enhance the usability of specific MMAs. They emphasize the rendering 
techniques, interaction paradigms and optimizing the system's perfor- 
mance. 



8. Balanced and simplistic visual design. Harmonious overall ap- 
pearance should consist of clear contrast between visual elements, bal- 
anced layout and informative colors. Visual elements should guide users 
gaze to important elements. Avoid visual clutter. 

In harmonious overall appearance all elements should work well together 
and complement each other (Tyner 2010). Harmony plays an important 
role in evaluating the overall appearance of mobile maps (Nival a & 
Sarjakoski 2005). According to Tyner (2010), balance, clarity and contrast 
are important in effective map design. The composition of a map should be 
balanced. Clarity of the map is mainly achieved through contrast, which can 
be created with opposites, such as dark and light. Visual clutter should be 
avoided. MacEachren (1995) states that the distinction between insignifi- 
cant and significant visual elements needs to be made clear in order to 
guide attention to specific details. Color is beneficial in the context of locat- 
ing and searching information (Van Laar 2001) and for grouping elements 
(Lidwell 2003). Krygier and Wood (2011) point out that colors should be 
more intense because of varying lighting conditions in use contexts of mo- 
bile maps. 

9. Recognizing, diagnosing and recovering from errors. Error 
messages should be expressed in plain language (no codes), precisely 
indicate the problem, and constructively suggest a solution. Indicate 
clearly the reasons for why the searched locations are not found. Save 
the user's previous searches for fast repetition. 

When errors occur, recovering from them should be straightforward. 
Haklay and Nivala (2010) state: "Actions that are reversible are important 
for relieving anxiety because it is clear to the user that errors can be undone 
and they should not feel that they will 'break' the system by one mistaken 
action." 

10. Offering help. Even though it is better if the system can be used with- 
out documentation, it may be necessary to provide help and documenta- 
tion. Provide both: fast guidance focused on the user's task and more 
detailed documentation with search functions. Pay attention to the un- 
derstandabi I ity of the hel p. 

Skarlatidou (2010) emphasize the careful design of help and documentation 
based on the purposes of the application and the user context. He points 
out that it is essential to offer instructions for the description and use of the 
tasks that are included in the application. Moreover, he states that as the 
majority of map application users are not familiar with the special termi- 
nol ogy of the fi el d, the vocabul ary and i nstructi ons shoul d be si mpl e. 



5. Conclusion 



We propose domain specific heuristics for evaluating the usability of 
MMAs. Four usability specialists tested initial, theoretically derived MMA 
heuristics versus the general Nielsen (1994) heuristics to evaluate the usa- 
bility of MMA. As a result, more usability problems were found with the 
initial domain-specific heuristics. Also, the initial heuristics were rated 
more applicable for the MMA domain. The proposed heuristics also include 
in-depth insights to the heuristic evaluation of visual mobile map design. 
Based on the results of this study it can be summarized that domain- 
specifi c usabi I ity heuri sti cs are needed i n order to properly eval uate M M As. 

A limitation in this study is the small amount of evaluators for testing the 
proposed heuristics. The results of HE are also known for being dependent 
on the evaluators (Hertzum & J acobsen 2003). Also, the general izabi I ity of 
the heuristics is uncertain at this stage, as the field of MMAs is wide. We 
will further validate the proposed MMA heuristics. Future steps will include 
testi ng the proposed heuristics with a larger data collection. 
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